Recent advances in depth mapping and object recognition have paved the way for immersive in-room augmented reality (AR) experiences. These are experiences that periodically or continuously monitor the state of a room, then show virtual objects in a user's field of vision. For example, existing hardware and applications enable scanning of an entire room to obtain a 3D model of the room geometry and objects or people within the room. Projectors mounted in the room can then display arbitrary content on various surfaces in the room. Further, voice commands, gestures, persons, faces, animals, furniture, and other objects can be recognized in real time, enabling applications to sense and respond to real-world events and objects.
As these and related capabilities become widespread, untrusted applications may have increased access to various forms of private data about the user and her surroundings. Examples of such data include room size and layout, number, size, activity, and gender of persons in the room, objects such as furniture, books, art, and visible text in the room, etc. Existing approaches to addressing privacy concerns related to exposure of such data typically take one of two different approaches.
For example one typical approach generally ignores privacy concerns and allows applications to obtain unrestricted access to raw sensor data for use in rendering content relative to the room based on that sensor data. This type of rendered content can be used in various gaming technologies, academic projects, vertically integrated augmented reality experiences, etc. Clearly one disadvantage of such applications is that users are expected to trust these applications with access to potentially sensitive data gathered by various sensors.
Another typical approach to addressing privacy concerns with respect to sensor data provides one or more application programming interfaces (API's) that gives developers and applications sharply restricted access to a subset of sensor information needed for a specific application. Such API's typically tightly control how applications render content. For example, one existing “augmented reality browser” on mobile phones includes an API that gives applications the ability to overlay textual or graphical annotations to “points of interest” visible on a display of the mobile phone. The application then decides where and how to show these annotations depending on whether the phone is pointed in the appropriate direction. For example, such applications may use a camera of the phone to image visible restaurants in the immediate vicinity of the user and then overlay a display of cuisine type, ratings, or price range onto the displayed image of the restaurants. While user privacy in such scenarios has increased protections by giving untrusted applications access only to a narrow API, each narrow API supports only a limited class of applications, and is custom tailored for each application. As a result, these types of API's tend to have limited usefulness.